CU Amiga Super CD-ROM 6

home *** CD-ROM | disk | FTP | other *** search

/ CU Amiga Super CD-ROM 6 / CU Amiga Magazine's Super CD-ROM 06 (1996)(EMAP Images)(GB)(Track 1 of 4)[!][issue 1997-01].iso / cucd / online / fidonetts / fsc-0072.001 < prev next >

Wrap

Text File | 1993-02-21 | 112KB | 1,906 lines

Document: FSC-0072 Version: 001 Date: 21-Feb-1993 The HYDRA file transfer protocol Joaquim H. Homrighausen and Arjen G. Lentz Status of this document: This FSC suggests a proposed protocol for the FidoNet(r) community, and requests discussion and suggestions for improvements. Distribution of this document is subject to the restrictions listed below. Fido and FidoNet are registered marks of Tom Jennings and Fido Software. --------------------------------------------------------------------- Copyright 1991-1993 Joaquim H. Homrighausen. All rights reserved. Copyright 1991-1993 Lentz Software Development. All rights reserved. --------------------------------------------------------------------- Restrictions ===================================================================== You are granted a license to implement the HYDRA file transfer protocol, HYDRA hereafter, in your own programs and/or use the sample source code and adapt these to your particular situation and needs; subject to the following conditions: o You must refer to it as the HYDRA file transfer protocol, and you must give credit to the authors of HYDRA in any information screens or literature pertaining to your programs that contains other such information (credits, your own copyrights, etc.). o HYDRA will always remain backwards compatible with previous revisions. HYDRA allows for expansion of its features without interfering with previous revisions. It is, however, important that different people do not expand the protocol in different directions. We therefore ask you to contact us if you have any needs/ideas regarding HYDRA, so development can be synchronized and beneficial to all. o If your implementation cannot converse with past or future revisions as supplied by us, then you must refer to it as "HYDRA derived", or as "a variation of HYDRA", or words to that effect. Permission is hereby granted to the FTSC (FidoNet Technical Standards Committee) and other technical organisations to republish this document in its entirety. Librarians may change the title page and page headers to match their library format as long as all copyrights and body text remain unaltered. The original document name and source must be mentioned in any republished versions of this document. No organization, company, person, or other being may impose any fees for any reason for providing this document. This document may not be sold or otherwise transferred for personal or company gain under any circumstances. Disclaimer ===================================================================== This information is provided "as is" and comes with no warranties of any kind, either expressed or implied. There is no support available for this package. It's intended to be used by programmers and developers. In no event shall the authors be liable to you or anyone else for any damages, including any lost profits, lost savings or other incidental or consequential damages arising out of the use or inability to use this information. Revision timestamps ===================================================================== 001 0x2b1aab00 Dec 01, 1992 Introduction ===================================================================== This document will not attempt to convince the reader that HYDRA is of value to him/her or that it is better than other file transfer protocols, it will simply describe the protocol. Just to get it out of the way, HYDRA is not the ultimate file transfer protocol. The authors do, however, feel that it offers an significant improvement over those file transfer protocols available today. HYDRA is a bi-directional protocol with the ability to receive and send files simultaneously. There are other bi-directional file transfer protocols, but to the authors' knowledge no public specifications exist. HYDRA owes much to Zmodem and its designer, Chuck Forsberg as well as to Janus, designed by Rick Huebner. We would like to think of HYDRA as a combination of both with a few extra options installed. The basic concept of a bi-directional file transfer protocol is simple. Both data channels are utilized to transmit and receive files simultaneously. I.e. two 100 kb files can be exchanged between two parties in the time it takes a fully streaming uni-directional file transfer protocol to transmit one of the files. Protocol design ===================================================================== The ultimate goal when designing HYDRA was to design a protocol that is as simple and robust as possible; complexity increase the problem of faulty implementations. The obvious function of a file transfer protocol is to transport a collection of data from its source to its destination as efficient possible and without jeopardizing the integrity of the data. The lack of data compression and lost packet management (as used in Kermit and Super Kermit) is intentional. The authors feel that this unnecessarily increases the complexity of the protocol. While HYDRA performs to its best on full duplex links, it should be possible to use it on links using proprietary protocols such as the US Robotics HST protocol which features one 14.4 kbps data channel and one 450 bps back channel. The protocol design should be flexible enough for future enhancements while maintaining backward compatibility. Protocol requirements and restrictions ===================================================================== HYDRA require that the link can handle ASCII character 24 (DLE) as well as all ASCII characters in the range 32 through 126. All other characters can be escaped or encoded by the protocol as required by the link. Capability of the computer to perform simultaneous serial I/O as well as simultaneous serial I/O combined with disk access is preferred, but can be circumvented by opting for windowed transmission instead of full streaming. HYDRA calls for the ability to check whether there is anything in the serial input buffer (i.e. "peek-ahead"), but it doesn't mind if it has to wait for a second if there is no data available (using for instance the UNIX alarm() mechanism). The protocol is extremely tolerant with timeouts (i.e. satellite or network delays) while still maintaining maximum reliability, robustness, and throughput. Terms and definitions ===================================================================== A BYTE An 8-bit unsigned character. A WORD A 16-bit unsigned integer. A DWORD A 32-bit unsigned integer. A LONG A 32-bit SIGNED integer. FILE OFFSETS (position) A long. NUL The ASCII character 0. BS The ASCII character 8. CR The ASCII character 13. XOFF The ASCII character 17. XON The ASCII character 19. H_DLE The HYDRA link escape character, ASCII 24 (^X). SP or SPACE The ASCII character 32. UNIX timestamp A specific time and date expressed as the number of seconds since midnight, January 1st, 1970. All UNIX timestamps used in HYDRA are expressed in local time. Multi-byte items are transmitted in "low-byte first" order, so big- endian CPUs (like 680xx) need to do some byteswapping, depending on the implementation. Values preceded by '0x' are in hexadecimal notation (base 16, 0..9 a..f). All values transmitted in hexadecimal notation must be converted to lowercase characters and left-padded to their full size with '0' prior to transmission. E.g. a WORD with the value 255 (decimal) is expressed as 00ff. A LONG with the value 255 (decmial) is expressed as 000000ff. In formulas, "AND" means bitwise AND, "XOR" means bitwise Exclusive OR, "NOT" is ones complement (i.e. all zeros become ones, all ones become zeros). The ">>" is a shift operation to the right, "R >> 3" means shift R three bits to the right. General packet format ===================================================================== All data exchange is done with framed packets protected by 16 or 32 bit CRC values appended to the packet data and packet type (low- byte first). The only exception to this is the cancel sequence of 5 consecutive H_DLE characters. All packets except those with the type DATA are followed by a CR (ASCII 13) to help get through some buffered environments and aid possible debugging and/or tracing. If requested by the other side in its INIT packet, packets can also be prefixed by a specific data string which can include NULs, delays or break signals. Refer to the section on the INIT packet for more information. Format of unframed packet +------------------------------------------+ ~ Zero or more bytes packet dependent data ~ +------------------------------------------+ | Packet type byte | +------------------------------------------+ | CRC-16/32 of packet data and packet type | +------------------------------------------+ Table of packet types +--------+---------+-----+--------------------------------+ |Name |Character|ASCII|Description | +--------+---------+-----+--------------------------------+ |START | 'A' | 65 |Startup sequence | |INIT | 'B' | 66 |Session initialisation | |INITACK | 'C' | 67 |Response to INIT packet | |FINFO | 'D' | 68 |File information | |FINFOACK| 'E' | 69 |Response to FINFO packet | |DATA | 'F' | 70 |File data packet | |DATAACK | 'G' | 71 |File data position ACK packet | |RPOS | 'H' | 72 |Reposition request packet | |EOF | 'I' | 73 |End of file packet | |EOFACK | 'J' | 74 |Response to EOF packet | |END | 'K' | 75 |End of session | |IDLE | 'L' | 76 |Idle (just saying I'm alive) | |DEVDATA | 'M' | 77 |Data to specified device (1)| |DEVDACK | 'N' | 78 |Response to DEVDATA packet (1)| +--------+---------+-----+--------------------------------+ (1) Support for DEVDATA and DEVDACK types is optional and indicated in INIT state of a HYDRA session. Format of framed packet +----------------------+--------------------+ | H_DLE |Packet format byte | +----------------------+--------------------+ ~ Encoded packet ~ +----------------------+--------------------+ | H_DLE |End of framed packet| +----------------------+--------------------+ Table of packet formats +----+---------+-----+--------------------------------+ |Name|Character|ASCII|Description | +----+---------+-----+--------------------------------+ |END | 'a' | 97 |End of framed packet | |BIN | 'b' | 98 |Binary packet | |HEX | 'c' | 99 |Hex encoded packet | |ASC | 'd' | 100 |Shifted 7-bit encoded packet (1)| |UUE | 'e' | 101 |UUencoded packet (1)| +----+---------+-----+--------------------------------+ (1) Support for ASC and/or UUE formats is optional and indicated in the INIT state of a HYDRA session. Packet sender and receiver state charts --------------------------------------------------------------------- TXPKT (Sender) +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+-+----------------------------+--------------------------+----------+ |Begin |1|pkttype == START or |format = HEXPKT |Format | | | |pkttype == INIT or | | | | | |pkttype == INITACK or | | | | | |pkttype == END or | | | | | |pkttype == IDLE | | | | +-+----------------------------+--------------------------+----------+ | |2|Escape 8th bit (7 bit link) | |Coding | | +-+----------------------------+--------------------------+----------+ | |3|else (no spc.pkt, 8bit link)|format = BINPKT |Format | +--------+-+----------------------------+--------------------------+----------+ |Coding |1|escape all control chars & |format = UUEPKT |Format | | | |UUENCODED packets allowed | | | | +-+----------------------------+--------------------------+----------+ | |2|ASCII packets allowed |format = ASCPKT |Format | | +-+----------------------------+--------------------------+----------+ | |3|7 bit link & |format = HEXPKT |Format | | | |escape all control chars & | | | | | |UUE/ASC pkts not allowed | | | +--------+-+----------------------------+--------------------------+----------+ |Format | |Append format byte to data|CRC | +--------+-+----------------------------+--------------------------+----------+ |CRC |1|format != HEXPKT & |Calc CRC-32 (data,pkttype)|Encode | | | |CRC-32 allowed |Append one's complement of| | | | | |CRC to data, lowbyte first| | | +-+----------------------------+--------------------------+----------+ | |2|else (HEXPKT or no CRC-32) |Calc CRC-16 (data,pkttype)|Encode | | | | |Append one's complement of| | | | | |CRC to data, lowbyte first| | +--------+-+----------------------------+--------------------------+----------+ |Encode |1|format == BINPKT |BIN escape databuf |Prefix | | +-+----------------------------+--------------------------+----------+ | |2|format == HEXPKT |HEX encode databuf |Prefix | | +-+----------------------------+--------------------------+----------+ | |3|format == ASCPKT |ASC encode/escape databuf |Prefix | | +-+----------------------------+--------------------------+----------+ | |4|format == UUEPKT |UUE encode databuf |Prefix | +--------+-+----------------------------+--------------------------+----------+ |Prefix |1|No more prefix characters | |Transmit | | +-+----------------------------+--------------------------+----------+ | |2|Prefix character ASCII 221 |Send 1 second break signal| | | +-+----------------------------+--------------------------+----------+ | |3|Prefix character ASCII 222 |1 second delay | | | +-+----------------------------+--------------------------+----------+ | |4|Prefix character ASCII 223 |Transmit NUL (ASCII 0) | | | +-+----------------------------+--------------------------+----------+ | |5|else (any other character) |Transmit character | | +--------+-+----------------------------+--------------------------+----------+ |Transmit| |Transmit H_DLE,format byte|Suffix | | | |Transmit encoded buffer | | | | |Transmit H_DLE,pktend byte| | +--------+-+----------------------------+--------------------------+----------+ |Suffix |1|pkttype != DATA & |Transmit CR,LF (ASC 13,10)|Done | | | |pktformat != BINPKT | | | | +-+----------------------------+--------------------------+----------+ | |2|else (pkttype == DATA or | |Done | | | | pktformat == BINPKT) | | | +--------+-+----------------------------+--------------------------+----------+ RXPKT (Receiver) +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Reset | |rxdle = 0 |NextByte | | | |format = 0 | | | | |pktlen = 0 | | +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|User wishes to abort session|Report reason for abort |Abort | | | |or carrier lost | | | | +-+----------------------------+--------------------------+----------+ | |2|Byte available in inputbuf | |StripIn | | +-+----------------------------+--------------------------+----------+ | |3|braintimer expired |Report braindead situation|Abort | | +-+----------------------------+--------------------------+----------+ | |4|Any other timer expired |Tell responsible party | | +--------+-+----------------------------+--------------------------+----------+ |StripIn |1|Escape 8th bit (7 bit link) |c = c AND 0x7f (strip 8th)|StripC | | +-+----------------------------+--------------------------+----------+ | |2|else (8 bit link) | |StripC | +--------+-+----------------------------+--------------------------+----------+ |StripC |1|Escape ctlchars with 8th set|n = c AND 0x7f (strip 8th)|Process | | +-+----------------------------+--------------------------+----------+ | |2|else (let 8 bit ctl through)|n = c |Process | +--------+-+----------------------------+--------------------------+----------+ |Process |1|c == H_DLE |increment rxdle |DLE | | +-+----------------------------+--------------------------+----------+ | |2|Escape XON/XOFF & |Eat these |NextByte | | | |n == XON or n == XOFF | | | | +-+----------------------------+--------------------------+----------+ | |3|Escape all control chars & |Eat these |NextByte | | | |n < 32 or n == 127 | | | | +-+----------------------------+--------------------------+----------+ | |4|rxdle > 0 | |Escape | | +-+----------------------------+--------------------------+----------+ | |5|else (no eating or escaping)| |Store | +--------+-+----------------------------+--------------------------+----------+ |DLE |1|rxdle == 5 |Report remote wants abort |Abort | | +-+----------------------------+--------------------------+----------+ | |2|else (rxdle < 5) | |NextByte | +--------+-+----------------------------+--------------------------+----------+ |Escape |1|c == PKTEND | |PktEnd | | +-+----------------------------+--------------------------+----------+ | |2|c == BINPKT |format = BINPKT |PktStart | | +-+----------------------------+--------------------------+----------+ | |3|c == HEXPKT |format = HEXPKT |PktStart | | +-+----------------------------+--------------------------+----------+ | |4|c == ASCPKT |format = ASCPKT |PktStart | | +-+----------------------------+--------------------------+----------+ | |5|c == UUEPKT |format = UUEPKT |PktStart | | +-+----------------------------+--------------------------+----------+ | |6|else (normal escaped char) |c = c XOR 0x40 |Store | | | | |rxdle = 0 |Store | +--------+-+----------------------------+--------------------------+----------+ |Store |1|format == 0 |Garbage |NextByte | | +-+----------------------------+--------------------------+----------+ | |2|pktlen >= maximum |Pkt too long / lost PKTEND|Reset | | +-+----------------------------+--------------------------+----------+ | |3|else (fmt > 0 & len < max) |Append c to databuffer |NextByte | | | | |increment pktlen | | +--------+-+----------------------------+--------------------------+----------+ |PktStart| |rxdle = 0 |NextByte | | | |pktlen = 0 | | +--------+-+----------------------------+--------------------------+----------+ |PktEnd |1|format == 0 |End without start, garbage|Reset | | +-+----------------------------+--------------------------+----------+ | |2|format == BINPKT |(No more decoding needed) |CalcCRC | | +-+----------------------------+--------------------------+----------+ | |3|format == HEXPKT |ok = Decode HEXPKT |CheckDec | | +-+----------------------------+--------------------------+----------+ | |4|format == ASCPKT |ok = Decode ASCPKT |CheckDec | | +-+----------------------------+--------------------------+----------+ | |5|format == UUEPKT |ok = Decode UUEPKT |CheckDec | +--------+-+----------------------------+--------------------------+----------+ |CheckDec|1|ok (no errors during decode)| |CalcCRC | | +-+----------------------------+--------------------------+----------+ | |2|else (errors in decoding) |Bad encoding, ignore pkt |Reset | +--------+-+----------------------------+--------------------------+----------+ |CalcCRC |1|format != HEXPKT & |Calc CRC-32 over databuf |CheckCRC | | | |CRC-32 allowed |ok = (crc == 0xdebb20e3) | | | | | |pktlen = pktlen - 4 | | | +-+----------------------------+--------------------------+----------+ | |2|else (HEXPKT or no CRC-32) |Calc CRC-16 over databuf |CheckCRC | | | | |ok = (crc == 0xf0b8) | | | | | |pktlen = pktlen - 2 | | +--------+-+----------------------------+--------------------------+----------+ |CheckCRC|1|ok (CRC matched magic) |pkttype = last byte of buf|Reset | | | | |pktlen = pktlen - 1 | | | | | |Hand pkt to higher level | | | +-+----------------------------+--------------------------+----------+ | |2|else (CRC check failed) |Bad CRC, ignore packet |Reset | +--------+-+----------------------------+--------------------------+----------+ BIN packet format --------------------------------------------------------------------- The binary packet format require an 8-bit data channel. If requested by either side, one or more sets of control characters are escaped. In this case, when one of these characters appears in an unframed packet, a H_DLE is sent followed by the character XOR 0x40. The H_DLE character itself is always transmitted in this fashion. On the receiver side, if the character after a H_DLE is not one of the packet format bytes, this character is decoded using XOR 0x40 again. BINPKT Escaping +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Begin | |txlastc = 0 |NextByte | +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|No more bytes to process | |Done | | +-+----------------------------+--------------------------+----------+ | |2|Escape ctlchars with 8th set|n = c AND 0x7f (strip 8th)|Escape | | +-+----------------------------+--------------------------+----------+ | |3|else (let 8 bit ctl through)|n = c |Escape | +--------+-+----------------------------+--------------------------+----------+ |Escape |1|n == H_DLE |Output H_DLE |Output | | | | |c = c XOR 0x40 | | | +-+----------------------------+--------------------------+----------+ | |2|Escape XON/XOFF & |Output H_DLE |Output | | | |n == XON or n == XOFF |c = c XOR 0x40 | | | +-+----------------------------+--------------------------+----------+ | |3|Escape Telenet & |Output H_DLE |Output | | | |n == CR & |c = c XOR 0x40 | | | | |txlasc == '@' | | | | +-+----------------------------+--------------------------+----------+ | |4|Escape all control chars & |Output H_DLE |Output | | | |n < 32 or n == 127 |c = c XOR 0x40 | | | +-+----------------------------+--------------------------+----------+ | |5|else (any other character) | |Output | +--------+-+----------------------------+--------------------------+----------+ |Output | |Store c |NextByte | | | |txlastc = c | | +--------+------------------------------+--------------------------+----------+ HEX packet format --------------------------------------------------------------------- Supported by all implementations, this packet format is used in worst-case situations and upon startup of a session when it is not yet known what restrictions the line and the other side will place on the link. Packet types always transmitted in HEX format are: START, INIT, INITACK, IDLE, END. HEX format packets always use a 16-bit CRC. HEX packets assume a 7-bit link, escaping all control characters and filtering all control characters upon receipt. ASCII characters in the range 128-255 (high bit set) are encoded by first transmitting a backslash ('\') character (ASCII 92), followed by the character in two lowercase hex-digits (bits 4-7 in first digit, bits 0-3 in second). Uppercase hex-digits are not permitted. The backslash character itself is transmitted as two backslashes. ASCII characters in the range 0-31 and 127 (all control characters) are escaped with H_DLE in the same fashion as in binary (BIN) packets. Decoded byte 1 +------+ 76543210 +--++--+ Encoded h1 h2 HEXPKT Encoding/Escaping +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|No more bytes to process | |Done | | +-+----------------------------+--------------------------+----------+ | |2|High bit of c set |Output \ (backslash) | | | | | |Output hexdigit(c bit 4-7)| | | | | |Output hexdigit(c bit 0-3)| | | +-+----------------------------+--------------------------+----------+ | |3|c < 32 or c == 127 |Output H_DLE | | | | | |Output (c XOR 0x40) | | | +-+----------------------------+--------------------------+----------+ | |4|c == \ (backslash) |Output \ (backslash) | | | | | |Output \ (backslash) | | | +-+----------------------------+--------------------------+----------+ | |5|else (any other character) |Output c | | +--------+-+----------------------------+--------------------------+----------+ HEXPKT Decoding +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|No more bytes to process | |Done OK | | +-+----------------------------+--------------------------+----------+ | |2|c == \ (backslash) | |Escape | | +-+----------------------------+--------------------------+----------+ | |3|else (any other character) |Output c |Escape | +--------+-+----------------------------+--------------------------+----------+ |Escape |1|No more bytes to process |Premature end of data |Error | | +-+----------------------------+--------------------------+----------+ | |2|c == \ (backslash) |Output \ (backslash) | | | +-+----------------------------+--------------------------+----------+ | |3|c == lowercase hexdigit |Save c, move ptr to next |NextHex | | +-+----------------------------+--------------------------+----------+ | |4|else (all other characters) |Invalid character |Error | +--------+-+----------------------------+--------------------------+----------+ |NextHex |1|No more bytes to process |Premature end of data |Error | | +-+----------------------------+--------------------------+----------+ | |2|c == lowercase hexdigit |Output (1st << 4 OR 2nd) |NextByte | | +-+----------------------------+--------------------------+----------+ | |3|else (all other characters) |Invalid character |Error | +--------+-+----------------------------+--------------------------+----------+ ASC packet format --------------------------------------------------------------------- Support of this packet format is optional and signalled in the INIT packet with the ASC flag in the "Supported options" field. 8-bit data is transformed into 7-bit data by a simple shift operation. Each byte is inserted at the top of a shift register, the lower seven bits are moved out. So seven 8-bit bytes are encoded into eight 7-bit characters. The end of the packet is padded by a maximum of six bits of 0 to make the number of bits a multiple of seven and thereby creating complete characters (so the receiver stops decoding when there are less than seven bits left). The output can contain control characters, so if escaping of these characters is required, this is done as in BIN packets using the H_DLE method. Decoded byte 7 byte 6 byte 5 byte 4 byte 3 byte 2 byte 1 +------++------++------++------++------++------++------+ 76543210765432107654321076543210765432107654321076543210 +-----++-----++-----++-----++-----++-----++-----++-----+ Encoded c8 c7 c6 c5 c4 c3 c2 c1 ASCPKT Encoding/Escaping +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Reset | |n = 0 (16 bit wide!) |NextByte | | | |bitshift = 0 | | +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|No more bytes to process | |Flush | | +-+----------------------------+--------------------------+----------+ | |2|else (more bytes to process)|n = n OR (c << bitshift) |Shift | | | | |BINPKT escape (n & 0x7f) | | | | | |n = n >> 7 | | | | | |increment bitshift | | +--------+-+----------------------------+--------------------------+----------+ |Shift |1|bitshift == 7 |BINPKT escape (n & 0x7f) |Reset | | +-+----------------------------+--------------------------+----------+ | |2|else (bitshift < 7) | |NextByte | +--------+-+----------------------------+--------------------------+----------+ |Flush |1|bitshift > 0 |BINPKT escape (n & 0x7f) |Done | | +-+----------------------------+--------------------------+----------+ | |2|else (bitshift == 0) | |Done | +--------+-+----------------------------+--------------------------+----------+ ASCPKT Decoding +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Begin | |n = 0 (16 bit wide!) |NextByte | | | |bitshift = 0 | | +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|No more bytes to process | |Done OK | | +-+----------------------------+--------------------------+----------+ | |2|else (more bytes to process)|c = c AND 0x7f |Shift | | | | |n = n OR (c << bitshift) | | | | | |bitshift = bitshift + 7 | | +--------+-+----------------------------+--------------------------+----------+ |Shift |1|bitshift >= 8 |Output (n AND 0xff) |NextByte | | | | |n = n >> 8 | | | | | |bitshift = bitshift - 8 | | | +-+----------------------------+--------------------------+----------+ | |2|else (bitshift < 8) | |NextByte | +--------+-+----------------------------+--------------------------+----------+ UUE packet format --------------------------------------------------------------------- Support of this packet format is optional and signalled in the INIT packet with the UUE flag in the "Supported options" field. The 8-bit data is transformed into printable ASCII using the UUENCODE algorithm. Three 8-bit bytes are encoded into four printable ASCII characters. This done by taking the bottom six bits left and adding '!' (ASCII 33) to move this character value into printable ASCII range. The end of the packet is padded by a maximum of five bits of 0 to make the number of bits a multiple of six and thereby creating complete characters (so the receiver stops decoding when there are less than six bits left). The output of this coding scheme does not need any further escaping before transmission. Decoded byte 3 byte 2 byte 1 +------++------++------+ 765432107654321076543210 +----++----++----++----+ Encoded c4 c3 c2 c1 UUEPKT Encoding +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|Less than three bytes left | |Flush | | +-+----------------------------+--------------------------+----------+ | |2|else (three or more left) |UUE(in[0]>>2) | | | | | |UUE(in[0]<<4 OR in[1]>>4) | | | | | |UUE(in[1]<<2 OR in[2]>>6) | | | | | |UUE(in[2]) | | | | | |(UUE: (c AND 0x3f) + '!') | | +--------+-+----------------------------+--------------------------+----------+ |Flush |1|No more bytes left | |Done | | +-+----------------------------+--------------------------+----------+ | |2|One byte left |UUE(in[0]>>2) |Done | | | | |UUE(in[0]<<4) | | | +-+----------------------------+--------------------------+----------+ | |3|Two bytes left |UUE(in[0]>>2) |Done | | | | |UUE(in[0]<<4 OR in[1]>>4) | | | | | |UUE(in[1]<<2) | | +--------+-+----------------------------+--------------------------+----------+ UUEPKT Decoding +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+-+----------------------------+--------------------------+----------+ |NextByte|1|Less than four bytes left | |Flush | | +-+----------------------------+--------------------------+----------+ | |2|else (four or more left) & |UD(i[0])<<2 OR UD(i[1])>>4| | | | |(c AND 0x7f) is in UUE range|UD(i[1])<<4 OR UD(i[2])>>2| | | | | |UD(i[2])<<6 OR UD(i[3]) | | | | | |(UD: (c - '!') AND 0x3f) | | | +-+----------------------------+--------------------------+----------+ | |3|else (all other characters) |Invalid character(s) |Error | +--------+-+----------------------------+--------------------------+----------+ |Flush |1|No bytes left or | |Done OK | | | |Less than two bytes left | | | | +-+----------------------------+--------------------------+----------+ | |2|Two bytes left & |UD(i[0])<<2 OR UD(i[1])>>4|Done OK | | | |(c AND 0x7f) is in UUE range| | | | +-+----------------------------+--------------------------+----------+ | |3|Three bytes left & |UD(i[0])<<2 OR UD(i[1])>>4|Done OK | | | |(c AND 0x7f) is in UUE range|UD(i[1])<<4 OR UD(i[2])>>2| | | +-+----------------------------+--------------------------+----------+ | |4|else (all other characters) |Invalid character(s) |Error | +--------+-+----------------------------+--------------------------+----------+ START packet (HEX format) --------------------------------------------------------------------- This packet is sent to tell the remote to initiate a HYDRA session. The complete framed packet as transmitted looks like: ASCII values 24 99 65 92 102 53 92 97 51 24 97 +-------+---+---+---+---+---+---+---+---+-------+---+ Characters | H_DLE | c | A | \ | f | 5 | \ | a | 3 | H_DLE | a | +-------+---+---+---+---+---+---+---+---+-------+---+ Applications may scan for this sequence to automatically start HYDRA when the remote transmits this packet (AutoStart). Prior to the START packet, a special string is transmitted to enable remote starting from a command prompt, hydra<CR> (the word hydra in lowercase): ASCII values 104 121 100 114 97 13 +---+---+---+---+---+----+ Characters | h | y | d | r | a | CR | +---+---+---+---+---+----+ The special string combined with the START packet is transmitted at five second intervals until either a START or INIT packet is received from the remote, or the maximum number of retries is reached. Any other packet types received in this stage must be ignored as they could be remains of a previous session. INIT packet (HEX format) --------------------------------------------------------------------- The INIT packet contains file transfer session options. The remote acknowledges this packet by returning an INITACK packet. INIT packet data +---------------------------------------+ ~ Application ID string, NUL terminated ~ +---------------------------------------+ ~ Supported options, NUL terminated ~ +---------------------------------------+ ~ Desired options, NUL terminated ~ +---------------------------------------+ | Desired transmitter window size or 0 | +---------------------------------------+ | Desired receiver window size or 0 | +---------------------------------------+ ~ Other general options, NUL terminated ~ +---------------------------------------+ ~ Packet prefix string, NUL terminated ~ +---------------------------------------+ The application ID string contains a printable ASCII string with the document revision date, product name, product revision number, and optionally the product serial number. The format of the string is: <RevDate><ProductName><,><ProductRevision>[<,><ProductSerial#>] RevDate is the UNIX timestamp (the hour, minute, and second portion is assumed to be zero), in hexadecimal notation, of the HYDRA document that the application is supposed to support. None of the following three fields should exceed thirty characters in length and must not contain any control characters (ASCII 0-31) or the comma character (ASCII 44). The field separator is a comma (ASCII 44) character. Capability flags XON Escape <XON> and <XOFF> characters. TLN Escape the <CR>@<CR> sequence (Telenet escape). CTL Escape ASCII characters 0-31 and 127. HIC Escape above three with high bit set. HI8 Escape ASCII characters 128-255 and strip the high bit. BRK Can transmit a break signal. ASC Can handle ASC packets. UUE Can handle UUE packets. C32 Can receive packets with 32-bit CRC error detection. DEV Can receive device packets. FPT Can receive filenames with paths. Capability flags are always three characters long, in uppercase, and seperated by a comma character (ASCII 44). Please note that the first five flags must be supported by all applications that implement the HYDRA specifications. The "Supported options" string contain the capability flags of the options that the application support. The "Desired options" string contain the capability flags of the options that the application would like to use/enable for the session. Some flags do not have to be specified in both strings. E.g. if the C32 flag is present in the "Supported options" string and the remote system indicates support for the same flag, 32-bit CRC error detection will be used. An application may not ask for an option it does not support. Escaping certain characters or bits also means filtering any occurrence of them in the incoming data stream. At the start of a session, it is assumed that the first five capability flags are in effect, i.e. the high bit is stripped off every received character and all control characters are filtered out. The "Desired transmitter/receiver window size" fields are long integers expressed in hexadecimal notation. With these options each side tells the other to use window management of the requested size when transmitting file data, instead of using full streaming (0). The window setting is completely seperate for both directions. If one side requests a smaller window size than the other, that smaller size will be used for that direction; also, a window of any size takes precedence over no window (0). Please note that the terms 'transmitter' and 'receiver' used for the fields in the INIT packet are from the view of the side transmitting that packet, so the other side should merge the 'transmitter' window field from the received INIT packet with its own 'receiver' window field. The "General options" string currently has no other fields than "Desired tx/rx window size"; the string is NUL terminated. The packet prefix string is normally empty, but may be provided by the remote if required. The maximum length of a packet prefix string is 30 characters. All characters should be transmitted as specified, with the following exceptions: Table of special packet prefix string chars +-----+--------------------------------------+ |ASCII|Description | +-----+--------------------------------------+ | 221 |Transmit a break signal for one second| | 222 |Delay one second before next character| | 223 |Transmit a NUL (ASCII 0) character | +-----+--------------------------------------+ INITACK packet (HEX format) --------------------------------------------------------------------- The INITACK packet is used to acknowledge the receipt of the remote's INIT packet. Duplicate INIT packets should be acknowledged too, as the remote may have missed previous INITACK packets; the reception of such a duplicate packet should not however reset the braindead timer, as it does not mean a change of state and is not actual file data. FINFO packet --------------------------------------------------------------------- File information packet, sent to notify the remote that another file is to be transmitted, or to signal end of batch. After the FINFO packet has been transmitted, the timer is set to the normal timeout value. The sender then waits for an FINFOACK packet from the remote or for the timer to expire. In the event of a timeout, the transmit/ wait sequence is repeated with half the normal timeout value until the maximum number of retries has been reached. FINFO packet data +-------------------------------------+ ~ File information, NUL terminated ~ +-------------------------------------+ File information End of batch +-----------------+ +-----------------+ | Timestamp or 0 | | NUL | +-----------------+ +-----------------+ | Filesize or 0 | +-----------------+ | Reserved (0) | +-----------------+ | Transaction # | +-----------------+ | File count or 0 | +-----------------+ ~ Short filename ~ +-----------------+ ~ Real filename ~ +-----------------+ | NUL | +-----------------+ End of batch is signalled by an empty string (only the terminating NUL). The first five fields are long integers expressed in hexadecimal notation. Timestamp is a UNIX timestamp representing the creation time of the file. If the creation time is not known, this is zero. Filesize is the size of the file in bytes. If the size of the file is not known, this is zero. This field should not be used as an exact measure of the size of the file. It is safe to assume that you should not receive less data than specified in this field, but the file may grow while it is being transferred (e.g. the result of a background process). Transaction # is a unique number for each set of files being sent during the session. This is primarily used to allow the receiving application to group several files together and store them in specific directories as a result of automated file requests. If the file being sent is not a result of an automated file request, this field must be set to zero. File count is the session file counter. For the first file in a session, this field contains the total number of files to be sent during the session; for subsequent files, it contains the file number in the session, starting with two (2). If the total number of files is not known, this field contains zero for all files. The first filename field must be specified in lowercase characters. It must conform to MS-DOS filename conventions and not exceed 12 characters in length (excluding the terminating NUL character). The second field, real filename, is the actual filename on the sending system. If this field is not present, the short filename field is used. No directory paths may be specified in the short filename. Directory paths may be specified in the real filename field if the "Desired options" of the receiver contains FPT. If the real filename field contains a path, it may include any ASCII character in the range 32 (0x20) through 255 (0xff) with \ characters translated to /. A drive specifier may be present in the <Drive>: (e.g. c:) format. If both the short and real filename fields are present, they are separated by a NUL character. There is never more than one NUL character terminating the packet. FINFOACK packet --------------------------------------------------------------------- Sent in response to an FINFO packet. If the FINFO packet contained file information, the FINFOACK packet is also used to instruct the remote how to proceed with the transfer. FINFOACK packet data +---------------------------------------+ | Long file offset, special code, or 0 | +---------------------------------------+ The only data in this packet is a long integer. In response to an an end of batch FINFO packet, the file offset is set to zero (0). In all other cases, file offset is one of the following: File offsets and special values +------+----------------------------------------------+ |Offset|Description | +------+----------------------------------------------+ | >=0 |Seek to specified offset and start sending (1)| | -1 |Already have file (2)| | -2 |Send file during another batch (not now) | +------+----------------------------------------------+ (1) This can only be something other than zero if the FINFO packet specified a filesize other than zero (i.e. the size of the file is known to the receiver). (2) The sending application should consider the file as having been sent successfully. This is primarily used to prevent duplicate files from being transmitted. DATA packet --------------------------------------------------------------------- Packet containing actual file data. DATA packet +-----------------------------------------+ | Long file offset of file data block | +-----------------------------------------+ ~ Variable length data block 0-2048 bytes ~ +-----------------------------------------+ If the file offset corresponds with what is expected by the receiver, the data block is saved and the file position increased accordingly. If the file offset is not correct, DATA packets may have been lost or failed the CRC check. Bad packets are ignored because it is not certain that the bad packet was an actual DATA packet and not some other type of packet. The file offset comparison is therefore the only way to find out about lost or bad data. When a bad data packet is detected, an RPOS packet is transmitted by the receiver to force the sender to seek to the desired file offset and resume transmission from it. After transmitting the RPOS packet, the receiver initializes a timer and continues to monitor received DATA packets while comparing their file offset with its desired offset. If the offset of a newly received DATA packet is greater than the offset received in the last DATA packet prior to transmitting the RPOS packet, the sender has not yet seen the RPOS packet, or the DATA packet was already in the data stream when the RPOS packet was transmitted. If the received offset matches the requested offset, the transfer is resumed, otherwise, a new RPOS packet is transmitted by the receiver and the timer restarted. If the timer expires, another RPOS packet is transmitted by the receiver. This is repeated until the maximum number of retries has been reached. If the receiver encounters more missing or invalid DATA packets at the same offset than it finds acceptable and it is not the originator of the session, it may decide to revert to a one-way transfer and wait with sending the remainder of its own files until the remote has transmitted its end of batch signal. It is possible that some hard- ware is not capable or well suited for a bi-directional file transfer involving large volumes of data (see description of the IDLE packet). DATAACK packet --------------------------------------------------------------------- Transmitted by the receiver with its current file offset after receiving a valid DATA packet. DATAACK packet data +------------------------+ | Long file offset | +------------------------+ This packet is only transmitted if there is a window in operation for that direction (selected in the INIT stage of the session), in which case the sender uses the DATAACK file offsets to manage its transmit window. If the sender's file offset is greater than or equal to the last DATAACK offset received plus the window size, no more data is transmitted by the sender until a DATAACK packet is received which allows the sender to proceed without exceeding the window size. While waiting for the DATAACK packet, the sender checks its timer and retry counter. If the timer expires before a valid DATAACK packet is received, the next DATA packet is transmitted, the retry counter incremented, and the timer restarted with half the normal timeout. This system ensures that the two sides do not end up waiting for each other in case packets are lost; the receiver will respond with either a DATAACK or RPOS packet. Receipt of a DATAACK packet does not reset the braindead timer. There are two windowing systems the receiver can use: sliding window or segmented streaming. If the receiver is capable of simultaneous serial and disk I/O, it will transmit a DATAACK packet for every received DATA packet, or every few DATA packets if it wants to be more economical with line capacity. Sliding window transmission is just a method of keeping the runahaid of the transmitter within reasonable limits (for sattelite or network links with long delays), thereby allowing for faster error recovery. Because of Hydra's tolerancy to delays and method of error recovery, sliding windows transmission is not normally required and full streaming can be used. If however the receiver is not capable of simultaneous serial and disk I/O, it will will process received DATA packets until the window size is reached (or slightly exceeded), write the received packets to disk, and then transmit one DATAACK packet to signal that it can receive the next segment of data. If the sender cannot handle simultaneous serial and disk I/O, it can apply the segmented streaming technique for reading data segments of the negotiated window size from disk. RPOS packet --------------------------------------------------------------------- Transmitted by the receiver to force the sender to seek to a specific position in the file and resume the transfer (as described above). The RPOS packet is also used by the receiver to skip a file once the transfer has been initiated. This is done by transmitting a RPOS packet with -2 as the desired offset and then waiting for a EOF packet with the same offset (-2). Once the EOF packet is received, the receiver responds to it by transmitting a EOFACK packet and then proceeds to wait for the next FINFO packet. RPOS packet dependent data +----------------------------------------+ | Long file offset | +----------------------------------------+ | Desired datablock size (word, 64-2048) | +----------------------------------------+ | Long RPOS packet ID | +----------------------------------------+ File offsets +------+-------------------------------------------+ |Offset|Description | +------+-------------------------------------------+ | >=0 |Seek to specified offset and resume sending| | -2 |Send file during another batch (not now) | +------+-------------------------------------------+ The desired data blocksize field tells the sender what blocksize to use when it resumes transmitting from the specified file offset. Each new RPOS packet should be given a different packet ID. This allows the sender to identify and ignore duplicate RPOS packets. The ID need not be sequential, but it must not have the same value as any other RPOS packet sent during the transmission of the same file. A RPOS ID value of zero (0) is not permitted. The same ID value is only used when sending multiple RPOS packets due to an expired RPOS packet timer as described above (DATA packet). EOF packet --------------------------------------------------------------------- Indicates that the end of the file has been reached by the sender. The packet is transmitted after the last DATA packet with file data. The EOF packet only contains one field which holds the current file offset of the sender (i.e. the actual size of the file). After the EOF packet has been transmitted, the timer is set to the normal timeout value. The sender then waits for an EOFACK packet from the remote or for the timer to expire. In the event of a timeout, the transmit/wait sequence is repeated with half the normal timeout value until the maximum number of retries has been reached. In the event that the receiver requests to skip the file by transmitting a RPOS(-2) packet (see RPOS packet), the EOF packet contains the same value (-2). If the sender wants to skip the file currently being transmitted, it issues an EOF packet with -2 as the offset value. EOF packets with an incorrect offset should be treated by the receiver as if it was an incorrect DATA packet (i.e. transmitting an RPOS packet). Accepted EOF packets are acknowledged by transmitting an EOFACK packet. EOF packet data +----------------------------------+ | Long file offset or special code | +----------------------------------+ File offsets and special value +------+----------------------------------------+ |Offset|Description | +------+----------------------------------------+ | >=0 |Final offset in file (size of file) | | -2 |Send file during another batch (not now)| +------+----------------------------------------+ EOFACK packet --------------------------------------------------------------------- Transmitted in response to an accepted EOF packet. After transmitting this packet, the receiver waits for the FINFO packet of the next file or end of batch. END packet (HEX format) --------------------------------------------------------------------- Once all files have been transmitted by both sides and no device packets remain to be transmitted, the end of session sequence is initiated. END packets are always sent in HEX format. Two END packets are transmitted and the transmit timer set to half the normal timeout. The application then waits for an END packet from the remote or for the transmit timer to expire. In the event of a timeout, the transmit/wait sequence is repeated until the maximum number of retries has been reached. At this point, the HYDRA session may be considered to be successful as both batches were completed. If an END packet is received before timeout, another three (3) END packets are transmitted and the protocol exits. Both sides need to transmit END packets and receive at least one from the remote. IDLE packet (HEX format) --------------------------------------------------------------------- The IDLE packet is used to let the remote know that the application is still alive. This is only applicable in uni-directional transfer mode to let the remote know that your application is still alive when it is receiving files, and after having transmitted an end of batch signal to the remote and not having any more files to send for the remainder of the session. When applicable, the IDLE packet is transmitted every 20 seconds. The remote resets its braindead timer upon receipt of an IDLE packet. If an application receives an IDLE packet while it is in a state where it is transmitting IDLE packets to the remote, something is wrong. This could occur if both sides have accidentally switched to one-way mode waiting for the remote to finish its batch. In this situation, one-way should be disabled to prevent a complete deadlock. Note that if both sides have finished their batch, the end of session sequence (see END packet) should be initiated. DEVDATA packet --------------------------------------------------------------------- Support for the DEVDATA and DEVDACK packets is optional and indicated in the INIT packet with the DEV flag in the "Supported options" field. The ID value is a long, different for each new device data packet sent. A value of zero (0) is not permitted. Only one DEVDATA packet may be transmitted at a time; the side issuing it then waits either for a timeout of the device transmit timer, or for a DEVDACK packet with the correct ID value to be received from the remote. If the timer expires before a correct DEVDACK packet is received, the DEVDATA packet is again transmitted, and the nnumber of device transmit retries incremented. If the maximum number of retries is reached, the HYDRA session is aborted; apparently the other is not functioning properly, or data is not getting through. In either case, the normal operation of the protocol (transferring files) will also fail. The name of the device to which the data is addressed is transmitted as an uppercase fixed-length three character NUL terminated string. There are two pre-defined device names as described below. If an unknown device name is specified, or a duplicate DEVDATA packet is received (one with the same ID value as a previously received and acknowledged DEVDATA packet), the packet is simply discarded after transmitting a DEVDACK packet with the corresponding ID value. DEVDATA and DEVDACK packets do not reset the braindead timer. They operate independently from the normal protocol. Device packets may only be transmitted after the initialization sequence, and before both sides have completed their batch. If a DEVDATA packet has not yet been acknowledged, the end of session sequence is delayed until a DEVDACK packet has been sent in response. DEVDATA packet dependent data +--------------------------------------------------+ | Long DEVDATA packet ID value | +--------------------------------------------------+ | 3 character uppercase device name, NUL terminated| +--------------------------------------------------+ ~ Variable length device data block (0-2048) ~ +--------------------------------------------------+ Predefined device names +---+-------------------------------------+ |Dev|Description | +---+-------------------------------------+ |MSG|Print data in protocol message window| |CON|Print data to user console | +---+-------------------------------------+ The MSG device may be used to notify the remote of protocol-specific issues, i.e. "One-way transfer mode". Such messages may be logged, but should not be considered to be machine-readable. The CON device may be used to implement a "chat" or conversation feature. This is a special case in which a session *can* be prolonged after end of batch, but not against the remote's will. While chat is enabled, there is no transition from the REND to the END transmitter state. When a CON device packet is transmitted in chat mode and the txstate is REND, the own braindead timer is reset. If the other side does initiate the end sequence by sending an END packet, the chat mode is immediately terminated and the session ended in a clean manner. If one side does not want to (continue) chat, and the other side does not comply, the one side will abort after a braindead timeout, so this chat system does not mean a security flaw. Each side is responsible for keeping the session going on his end until its own user has finished chatting. It is suggested that the software apply a timeout of say 1 minute to keyboard input, ending the chat automatically if the user stops typing but does not exit chat mode. Also, the chat mode should be initiated with a special key so that it can not erronously be started or prolonged. DEVDACK packet --------------------------------------------------------------------- Transmitted in response to a DEVDATA packet. The device data ID value must correspond to the ID of the previously received DEVDATA packet. DEVDACK packet data +---------------------------+ | Long device data ID value | +---------------------------+ DEVICE sender (devtxstate HTD_...) +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Begin | |devtxid = 0 |DONE | | | |reset devtxtimer | | +--------+-+----------------------------+--------------------------+----------+ |DONE |1|wish to send device data & |increase devtxid |DATA | | | |other side allows DEV pkts |devtxretries = 0 | | | | | |reset devtxtimer | | | +-+----------------------------+--------------------------+----------+ | |2|wish to send device data & |Tell calling function | | | | |other doesn't allow DEV pkts|it's not on... | | +--------+-+----------------------------+--------------------------+----------+ |DATA |1|devtxretries == 10 |Report too many errors |Abort | | +-+----------------------------+--------------------------+----------+ | |2|else (devtxretries < 10) |txpkt DEVDATA(id,dev,data)|DACK | | | | |devtxtimer = timeout | | +--------+-+----------------------------+--------------------------+----------+ |DACK |1|rxpkt DACK & |reset devtxtimer |DONE | | | |DACK(id) == devtxid | | | | +-+----------------------------+--------------------------+----------+ | |2|devtxtimer expired |Report devtx timeout |DATA | | | | |increase devtxretries | | +--------+-+----------------------------+--------------------------+----------+ DEVICE RECEIVER +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Begin | |devrxid = 0 |DONE | +--------+-+----------------------------+--------------------------+----------+ |DONE |1|rxpkt DEVDATA |txpkt DEVDACK(id) |CheckID | +--------+-+----------------------------+--------------------------+----------+ |CheckID |1|DEVDATA(id) != devrxid |devrxid = DEVDATA(id) |Process | | +-+----------------------------+--------------------------+----------+ | |2|else (apparent duplicate) | |DONE | +--------+-+----------------------------+--------------------------+----------+ |Process |1|DEVDATA(dev) == MSG |Print protocol message |DONE | | +-+----------------------------+--------------------------+----------+ | |2|DEVDATA(dev) == CON |Output to user console |DONE | | +-+----------------------------+--------------------------+----------+ | |3|DEVDATA(dev) == known&ok |Call processing routine |DONE | | +-+----------------------------+--------------------------+----------+ | |4|else (unknown/not-ok device)|One-way into bitbucket |DONE | +--------+-+----------------------------+--------------------------+----------+ Packet length and data block size --------------------------------------------------------------------- The maximum length of a source data block (i.e. raw, non processed input file or device data) is 2048 bytes. The maximum allowed length of the packet data is 2048 + 8 = 2056 bytes. The eight bytes are to provide sufficient room for the additional fields in the DATA and DEVDATA packets. Packetizing adds an additional three to five bytes. The maximum length of a framed packet being transmitted can be three times the size of its source data depending on what type of encoding scheme is used (ASC, HEX, UUE, BIN). The minimum length of a data block is 64 bytes. The block size of DATA packets is based on the physical (DCE) link speed and is established as follows: +---------+------------------+-------------------+ |DCE speed|Maximum block size|Starting block size| +---------+------------------+-------------------+ | 300 bps| 256 bytes | 256 bytes | | 1200 bps| 512 bytes | 256 bytes | | 2400 bps| 1024 bytes | 512 bytes | |>2400 bps| 2048 bytes | 512 bytes | +---------+------------------+-------------------+ The blocksize is initialized to the starting blocksize when a session is first started. After each kilobyte of file data transmitted, the blocksize is doubled until it reaches the maximum allowed blocksize. When the maximum allowed blocksize has been reached, the variable keeping track of how many bytes are needed to increase the blocksize is reset to zero. If a request for retransmission (RPOS packet) is received from the receiver, the blocksize is immediately set to that specified in the retransmission request. Every time this occurs, the number of bytes needed to double the blocksize is increased by 1024 with a maximum of of 8192 bytes. The end result is that more data has to be successfully transmitted before the blocksize is increased for each error that occurs. The length of a data block is dynamic and always in the range 0-2048 bytes. A data block is never padded. If there is insufficient data to fill a block of the current blocksize, the blocksize is adjusted to the amount of remaining data. The blocksize logic is not reset between files in a session. Timers and retry counters ===================================================================== Each process in the protocol (transmit, receive and device transmit) has its own timer and retry counter, and there is one overall braindead timer. Allowed are 10 tries, the braindead timeout is 120 seconds, and the other timeouts are based on the speed of the line and the state of the protocol. It can be calculated as (40960/DCE rate), with a minimum of 10 seconds and a maximum of 60 seconds. +---------+----------+----------+ |DCE speed|Timeout |Half | +---------+----------+----------+ | 300 bps|60 seconds|30 seconds| | 1200 bps|34 seconds|17 seconds| | 2400 bps|17 seconds| 8 seconds| |>2400 bps|10 seconds| 5 seconds| +---------+----------+----------+ If the output buffer is empty, the timeout value is halved. In general, this is the case if the number of tries is greater than zero and during the init and end sequences. These timeouts are not fatal situations, they just give the remote a reasonable amount of time to receive and respond to a packet before a retry occurs. Duplicate packets are always identified and ignored. A retry counter is reset if there is a change of state, or a reposition different from the previous file offset occurs. The braindead timer monitors useful data from the other side: a first response to a transmitted supervisiory packet, or a received packet with file data at the correct offset. Device packets and packets that do not signify any progress of the protocol do not affect the braindead timer. No other timers (such as one between characters in a packet) are necessary. Aborting a session ===================================================================== A session is aborted with five consequetive CAN (^X or ASCII 24) characters. Whenever a state table mentions "Abort", the following procedure is to be followed: Clear the output buffer and transmit eight CAN characters followed by ten BS (^H or ASCII 8) characters; wait a few seconds for the data to be transmitted to the remote, purge the input buffer and exit the protocol code. GENERAL sender (txstate HTX_...) +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Begin | |txretries = 0 |START | | | |reset txtimer | | | | |blksize = startblksize | | | | |goodbytes = 0 | | | | |goodneeded = 1024 | | | | |braintimer = 120 | | +--------+-+----------------------------+--------------------------+----------+ |START |1|txretries == 10 |Report too many errors |Abort | | +-+----------------------------+--------------------------+----------+ | |2|else (txretries < 10) |txstr AutoStart |SWAIT | | | | |txpkt START | | | | | |txtimer = 5 | | +--------+-+----------------------------+--------------------------+----------+ |SWAIT |1|rxpkt START or |txretries = 0 |INIT | | | |rxpkt INIT |reset txtimer | | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |2|txtimer expired |Report tx timeout |START | | | | |increment txretries | | +--------+-+----------------------------+--------------------------+----------+ |INIT |1|txretries == 10 |Report too many errors |Abort | | +-+----------------------------+--------------------------+----------+ | |2|else (txretries < 10) |txpkt INIT(linkinfo) |INITACK | | | | |txtimer = timeout/2 | | +--------+-+----------------------------+--------------------------+----------+ |INITACK |1|rxpkt INITACK |txretries = 0 |RINIT | | | | |reset txtimer | | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |2|txtimer expired |Report tx timeout |INIT | | | | |increment txretries | | +--------+-+----------------------------+--------------------------+----------+ |RINIT |1|rxstate != INIT | |NextFile | +--------+-+----------------------------+--------------------------+----------+ |NextFile|1|No files left? |Report end of batch |ToFName | | | | |Set NULL fileinfo | | | +-+----------------------------+--------------------------+----------+ | |2|Can access next file? |Set up fileinfo |ToFName | | +-+----------------------------+--------------------------+----------+ | |3|Can't access file? |Report access failure |NextFile | +--------+-+----------------------------+--------------------------+----------+ |ToFName | |txsyncid = 0 |FINFO | | | |txretries = 0 | | | | |reset txtimer | | +--------+-+----------------------------+--------------------------+----------+ |FINFO |1|txretries == 10 |Report too many errors |Abort | | +-+----------------------------+--------------------------+----------+ | |2|txretries > 0 |txpkt FINFO(fileinfo) |FINFOACK | | | | |txtimer = timeout/2 | | | +-+----------------------------+--------------------------+----------+ | |3|else (txretries == 0) |txpkt FINFO(fileinfo) |FINFOACK | | | | |txtimer = timeout | | +--------+-+----------------------------+--------------------------+----------+ |FINFOACK|1|NULL fname (end of batch) & |txtimer = idletimeout |REND | | | |rxpkt FINFOACK |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |2|rxpkt FINFOACK & |txpos = FINFOACK(pos) |DATA | | | |FINFOACK(pos) >= 0 |txretries = 0 | | | | | |txlastack = 0 | | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |3|rxpkt FINFOACK & |They already have file |NextFile | | | |FINFOACK(pos) == -1) |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |4|rxpkt FINFOACK & |Skip this file for now |NextFile | | | |FINFOACK(pos) == -2) |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |5|txtimer expired |Report tx timeout |FINFO | | | | |inrease txretries | | +--------+-+----------------------------+--------------------------+----------+ |DATA |1|rxstate != Done & |txtimer = idletimeout |XWAIT | | | |hdxlink == True | | | | +-+----------------------------+--------------------------+----------+ | |2|rxpkt DATAACK & |txlastack = DATAACK(pos) | | | | |DATAACK(pos) > txlastack | | | | +-+----------------------------+--------------------------+----------+ | |3|rxpkt RPOS & |Skip this file for now |SkipFile | | | |RPOS(pos) < 0 |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |4|rxpkt RPOS & |Report too many errors |Abort | | | |RPOS(id) == txsyncid & | | | | | |inc txretries == 10 | | | | +-+----------------------------+--------------------------+----------+ | |5|rxpkt RPOS & |txpos = RPOS(pos) | | | | |RPOS(id) != txsyncid & |txsyncid = RPOS(id) | | | | |RPOS(pos) >= 0 |txretries = 1 | | | | | |blksize = RPOS(blksize) | | | | | |goodbytes = 0 | | | | | |inc goodneeded + 1024 | | | | | |if (goodneeded > 8192) | | | | | | goodneeded = 8192 | | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |6|File seek/read error or |Skip this file for now |SkipFile | | | |user wishes to skip file | | | | +-+----------------------------+--------------------------+----------+ | |7|txwindow & |if (txretries > 0) |DATAACK | | | |txpos >= txlastack+txwindow | txtimer = timeout/2 | | | | | |else | | | | | | txtimer = timeout | | | +-+----------------------------+--------------------------+----------+ | |8|Enough room in output & |txpkt DATA(pos,data) | | | | |more filedata(blksize) to go|txpos += datalen | | | | | |inc goodbytes + datalen | | | | | |if goodbytes > goodneeded | | | | | | Store txpos,blksize | | | | | | blksize * 2 (max.2048) | | | +-+----------------------------+--------------------------+----------+ | |9|End of filedata reached | |EOF | +--------+-+----------------------------+--------------------------+----------+ |SkipFile| |txpos = -1 |EOF | | | |txretries = 0 | | +--------+-+----------------------------+--------------------------+----------+ |DATAACK |1|txretries == 10 |Report too many errors |Abort | | +-+----------------------------+--------------------------+----------+ | |2|rxpkt DATAACK & |txlastack = DATAACK(pos) |DATA | | | |DATAACK(pos) > txlastack & |txretries = 0 | | | | |txpos < DATAACK(pos) + txwin|reset txtimer | | | +-+----------------------------+--------------------------+----------+ | |3|rxpkt RPOS |Handle RPOS in state DATA | | | | | |but stay in this state | | | +-+----------------------------+--------------------------+----------+ | |4|txtimer expired |Report tx timeout |DATA | | | | |increment txretries | | +--------+-+----------------------------+--------------------------+----------+ |XWAIT |1|rxstate == Done |reset txtimer |DATA | | +-+----------------------------+--------------------------+----------+ | |2|rxpkt DATAACK & |txlastack = DATAACK(pos) | | | | |DATAACK(pos) > txlastack | | | | +-+----------------------------+--------------------------+----------+ | |3|rxpkt RPOS |Handle RPOS in state DATA | | | | | |but stay in this state | | | +-+----------------------------+--------------------------+----------+ | |4|rxpkt IDLE |hdxlink = False |DATA | | | | |reset txtimer | | | +-+----------------------------+--------------------------+----------+ | |5|txtimer expired |txpkt IDLE | | | | | |txtimer = idletimeout | | +--------+-+----------------------------+--------------------------+----------+ |EOF |1|txretries == 10 |Report too many errors |Abort | | +-+----------------------------+--------------------------+----------+ | |2|txretries > 0 |txpkt EOF(txpos) |EOFACK | | | | |txtimer = timeout/2 | | | +-+----------------------------+--------------------------+----------+ | |3|else (txretries == 0) |txpkt EOF(txpos) |EOFACK | | | | |txtimer = timeout | | +--------+-+----------------------------+--------------------------+----------+ |EOFACK |1|rxpkt EOFACK |braintimer = 120 |NextFile | | +-+----------------------------+--------------------------+----------+ | |2|rxpkt DATAACK & |txlastack = DATAACK(pos) | | | | |DATAACK(pos) > txlastack | | | | +-+----------------------------+--------------------------+----------+ | |3|rxpkt RPOS & |rxpos == -2 |EOF | | | |RPOS(pos) == -2 & | | | | | |rxpos != -2 | | | | +-+----------------------------+--------------------------+----------+ | |4|rxpkt RPOS & |Handle as in state DATA |DATA | | | |RPOS(pos) >= 0 | | | | +-+----------------------------+--------------------------+----------+ | |5|txtimer expired |Report tx timeout |EOF | | | | |increment txretries | | +--------+-+----------------------------+--------------------------+----------+ |REND |1|rxstate == DONE & |txretries = 0 |END | | | |devtxstate == DONE | | | | +-+----------------------------+--------------------------+----------+ | |2|txtimer expired |txpkt IDLE | | | | | |txtimer = idletimeout | | +--------+-+----------------------------+--------------------------+----------+ |END |1|txretries == 10 | |Abort | | +-+----------------------------+--------------------------+----------+ | |2|else (txretries < 10) |txpkt END (twice) |ENDACK | | | | |txtimer = timeout/2 | | +--------+-+----------------------------+--------------------------+----------+ |ENDACK |1|rxpkt END |txpkt END (thrice) |Done | | +-+----------------------------+--------------------------+----------+ | |2|txtimer expired |Report tx timeout |END | | | | |increment txretries | | +--------+-+----------------------------+--------------------------+----------+ GENERAL RECEIVER (rxstate HRX_...) +--------+------------------------------+--------------------------+----------+ |State |Predicate(s) |Action(s) |Next state| +--------+------------------------------+--------------------------+----------+ |Begin | |reset rxtimer |START | | | |lastrxdlen = startblksize | | | | |(tx handles braintimer) | | +--------+-+----------------------------+--------------------------+----------+ |INIT |1|rxpkt INIT & |txpkt INITACK |FINFO | | | |INIT(options) are compatible|Set options | | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |2|INIT(options) not compatible|Report link failure |Abort | +--------+-+----------------------------+--------------------------+----------+ |FINFO |1|rxpkt INIT (apparent dup) |txpkt INITACK | | | +-+----------------------------+--------------------------+----------+ | |2|rxpkt FINFO & |Report end of batch | | | | |FINFO(fileinfo) == Empty |txpkt FINFOACK |DONE | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |3|rxpkt FINFO & |do not want this file | | | | |we already have file |txpkt FINFOACK(-1) | | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |4|rxpkt FINFO & |Skip this file for now | | | | |open/diskspace error |txpkt FINFOACK(-2) | | | | | |braintimer = 120 | | | +-+----------------------------+--------------------------+----------+ | |5|rxpkt FINFO & |rxpos = resume offset |ToData | | | |file we want to resume | | | | +-+----------------------------+--------------------------+----------+ | |6|rxpkt FINFO & |rxpos = 0 |ToData | | | |new file for us | | | | +-+----------------------------+--------------------------+----------+ | |7|rxpkt EOF (apparent dup) |txpkt EOFACK | | +--------+-+----------------------------+--------------------------+----------+ |ToData | |txpkt FINFOACK(rxpos) |DATA | | | |rxsyncid = 0 | | | | |rxlastsync = 0; | | | | |rxretries = 0 | | | | |reset rxtimer | | | | |braintimer = 120 | | +--------+-+----------------------------+--------------------------+----------+ |DATA |1|rxpkt FINFO (apparent dup) |txpkt FINFOACK(rxpos) | | | +-+----------------------------+--------------------------+----------+ | |2|rxpkt DATA & |Store data | | | | |DATA(pos) == rxpos |rxpos += datalen | | | | | |rxretries = 0 | | | | | |rxlastsync = rxpos | | | | | |reset rxtimer | | | | | |braintimer = 120 | | | | | |if (rxwindow) | | | | | | txpkt DATAACK(rxpos) | | | +-+----------------------------+--------------------------+----------+ | |3|rxpkt DATA & |Report bad rxpos |BadPos | | | |DATA(pos) != rxpos | | | | +-+----------------------------+--------------------------+----------+ | |4|rxpkt EOF & |Close file, received ok |OkEOF | | | |EOF(pos) == rxpos | | | | +-+----------------------------+--------------------------+----------+ | |5|rxpkt EOF & |Close, save for resume |OkEOF | | | |EOF(pos) == -2 | | | | +-+----------------------------+--------------------------+----------+ | |6|rxpkt EOF & |Report bad EOF |BadPos | | | |EOF(pos) != rxpos | | | | +-+----------------------------+--------------------------+----------+ | |7|File write error or |Close, save for resume | | | | |user wishes to skip file |rxpos = -2 | | | +-+----------------------------+--------------------------+----------+ | |8|rxpkt IDLE & |braintimer = 120 | | | | |hdxlink == False | | | +--------+-+----------------------------+--------------------------+----------+ |BadPos |1|DATA/EOF(pos) <= rxlastsync |rxretries = 0 |Timer | | | | |reset rxtimer | | | | | |rxlastsync = pos | | | +-+----------------------------+--------------------------+----------+ | |2|DATA/EOF(pos) > rxlastsync |rxlastsync = pos |Timer | +--------+-+----------------------------+--------------------------+----------+ |Timer |1|rxtimer expired | |HdxLink | | +-+----------------------------+--------------------------+----------+ | |2|else (rxtimer not expired) | |DATA | +--------+-+----------------------------+--------------------------+----------+ |HdxLink |1|rxretries > 4 & |hdxlink = True |Retries | | | |txstate < REND & |rxretries = 0 | | | | |originator == False & | | | | | |hdxlink == False | | | | +-+----------------------------+--------------------------+----------+ | |2|else (above not the case) | |Retries | +--------+-+----------------------------+--------------------------+----------+ |Retries |1|inc rxretries == 10 |Report too many errors |Abort | | +-+----------------------------+--------------------------+----------+ | |2|rxretries == 1 |increase rxsyncid |RPos | | +-+----------------------------+--------------------------+----------+ | |3|else (rxretries > 1) | |RPos | +--------+-+----------------------------+--------------------------+----------+ |RPos | |lastrxdatalen/=2 (min.64) |DATA | | | |txpkt RPOS (rxpos, | | | | | lastrxdatalen,rxsyncid)| | | | |rxtimer = timeout | | +--------+------------------------------+--------------------------+----------+ |OkEOF | |txpkt EOFACK |FINFO | | | |reset rxtimer | | | | |braintimer = 120 | | +--------+-+----------------------------+--------------------------+----------+ |DONE |1|rxpkt FINFO (apparent dup) |txpkt FINFOACK(-2) | | +--------+-+----------------------------+--------------------------+----------+ HYDRA in FidoNet technology mailers ===================================================================== HYDRA is suitable for use in FidoNet mailers. It can be implemented for EMSI and FTS-6 mail sessions. The FTS-6 (YooHoo) capability bit for HYDRA is 0x0020 (DOES_HYDRA). The EMSI and IEMSI protocol capability flag is HYD. When utilizing HYDRA in a mail session, two complete batches are always performed. Little else differs from a normal FTS-6 ZedZap mail session. The first batch is used to transmit all mail, files, and file requests by both sides. The second batch is always performed, sending nothing if there are no file requests to honor. The data buffers are not purged between the two batches since HYDRA ignores any leftovers from the previous batch (END packets, etc.). To integrate HYDRA into an existing mailer, the same code used for the ZedZap session flow can be used, but instead of one transmit and one receive session, two transmit sessions (or batches) are used. When the HYDRA end of batch is initiated it will not be terminated until an end of batch has been received from the remote and the end of session sequence has been finished. Fido and FidoNet are registered marks of Tom Jennings and Fido Software. Error detection using CRCs ===================================================================== CRC (Cyclic Redundancy Check) values only provide their promised maximum error detection capability when properly applied, which involves calculating and transmitting low-bit first, presetting the CRC with all ones, and transmitting the ones' complement of the result. The receiver also initializes to all ones, processes all of the data AND the following CRC, and the result should match a "magic value" which is 0xf0b8 for the 16-bit CRC and 0xdebb20e3 for the 32-bit CRC. The easiest and fastest way to perform CRC calculations is by using a table that does the algorithm's shift-operations in 8-bits at a time. CRC-16 error detection --------------------------------------------------------------------- 16-bit CRC using the CCITT CRC-16 polynomial. This is the default at startup, and always used for HEX packets even if both sides are capable of handling 32-bit CRCs. This CRC-16 is not identical to the one used by the Xmodem and Zmodem file transfer protocols. The polynomial is the same (X^16+X^12+X^5+X^0 or 0x8408) but the bit-ordering is the opposite, and preconditioning and postconditioning is used as in 32-bit CRCs. This method is also used by the European version of X.25. The 16-bit CRC table is created as follows (pseudocode, the variable CRC16 and the table of 256 entries are 16-bit unsigned integers): FOR (i=0 TO 255) { CRC16=i FOR (N=1 TO 8) { IF (CRC16 AND 1) CRC16=(CRC16 >> 1) XOR 0x8408 ELSE CRC16=CRC16 >> 1 } CRC16TAB[i]=CRC16; } When processing data, each byte is run through the CRC calculation routine as follows (variable CRC stores the 16-bit CRC value/result, C is the next 8-bit char): CRC=CRC16TAB[(CRC XOR C) AND 0xff] XOR ((CRC>>8) AND 0x00ff) CRC-32 error detection --------------------------------------------------------------------- 32-bit CRC using the CCITT CRC-32 polynomial. Support of CRC-32 is optional and signalled in the INIT packet. If both sides indicate they can handle CRC-32, all packets except those transmitted in HEX format use this algorithm instead of CRC-16 to improve error detection. This CRC-32 is identical to the one used by the Zmodem protocol. The polynomial is (0xedb88320): X^32+X^26+X^23+X^22+X^16+X^12+X^11+X^10+X^8+X^7+X^5+X^4+X^2+X^1+X^0 The principal method of calculation, transmission, and checking is identical to CRC-16 as described above, but the "magic value" for 32-bit CRC is 0xdebb20e3. The CRC-32 table is created as follows (pseudocode, the variable CRC32 and the table of 256 entries are 32-bit unsigned integers): FOR (i=0 TO 255) { CRC32=i FOR (N=1 TO 8) { IF (CRC32 AND 1) CRC32 = (CRC32 >> 1) XOR 0xedb88320 ELSE CRC32 = CRC32 >> 1 } CRC32TAB[i] = CRC32; } When processing data, each byte is run through the CRC calculation routine as follows (variable CRC stores the 32-bit CRC value/result, C is the next 8-bit character): CRC=CRC32TAB[(CRC XOR C) AND 0xFF] XOR ((CRC>>8) AND 0x00ffffff) The authors ===================================================================== The authors can be reached at the following addresses: Joaquim H. Homrighausen Arjen G. Lentz 389, route d'Arlon Lentz Software Development L-8011 Strassen Langegracht 7B Luxembourg 3811 BT Amersfoort The Netherlands joho@ae.lu FidoNet 2:270/17 aglentz@fido.lu FidoNet 2:283/512 The name HYDRA ===================================================================== Hydra is a greek mythological creature (the watersnake). Like the Nemeic lion, Hydra is the daughter of the giant Typhon and the snake Echidna. She grew up in the marshes of Lerna near/in Argolis (Argos). There she ate entire herds of cattle and destroyed large cropfields. Later, she lived in caves on a hill near the spring of Amymone. Hydra is a monstrous large snake with nine heads: eight mortal ones, and one (the middle one) immortal. She was defeated and killed by Heracles (Hercules) - son of Zeus and Alcemene, grandson of Perseus - as the second of his twelve tasks, with the help of his cousin Iolaos. Every time he cut of one of the heads with his sword, two new heads grew in its place. So Iolaos scorched the wound of each cut off head with burning branches so the head couldn't grow on again. Heracles buried the last and immortal head under a stone nearby. He also dipped his arrows in Hydra's poisonous blood, thereafter the wounds caused by those arrows were incurable. Also star constellation (sign of the watersnake) in the equatorial zone. Also a type of sweetwater polip (the Hydroidea Thin, tubeshaped body can be full contracted, just like the six (or more) tentacles. There is no generation change, the gender-products grow directly on the body. The animals catch their prey with nettlecells, and are very verocious. They can be found in various stilstanding and flowing water and were first described by Anthonie van Leeuwenhoek in the year 1704. Also small island (spelled Idhra in modern Greek) of the Sargonic group in the Aegean Sea, just of the eastern tip of the Argolis peninsula of the Peloponesus (Attika). Its length (NE/SW) is 21 km, with a total area of 49,6 square km. Its highest point is 597 meters above sea level. Population of 2794 (latest census: 1981). Only one real city with the same name as the island. Once quite wooded and well watered, now denuded and dry, with almost no arable land (infertile limestone). Certain times of the year the people have to ship in water from the main land. Its Turkish name is Camliza which means "Place of Pines". References: Dutch: Oosthoeks Encyclopedie Grote Winklerprins Encyclopedie English: Encyclopaedia Brittannica --- end of "hydra.doc" ---